Interactive Gesture in Dialogue: a PTT Model

Gestures are usually looked at in isolation or from an intra-propositional perspective essentially tied to one speaker. The Bielefeld multi-modal Speech-AndGesture-Alignment (SAGA) corpus has many interactive gestures relevant for the structure of dialogue (Rieser 2008, 2009). To describe them, a dialogue theory is needed which can serve as a speechgesture interface. PTT (Poesio and Traum 1997, Poesio and Rieser submitted a) can do this job in principle, how this can be achieved is the main topic of this paper. As a precondition, the empirical research procedure from systematic corpus annotation via gesture typology to a partial ontology for gestures is described. It is then explained how PTT is extended to provide an incremental modelling of speech plus gesture in an assertion-acknowledgement adjacency pair where grounding between dialogue participants is obtained through gesture.